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Status 

1)13 Responsive to communication(s) filed on 21 August 2007 . 
2a)S This action is FINAL. 2b)n This action is non-final. 

3) 0 Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 . 453 O.G. 213. 
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4) 13 Claim(s) 1-22 is/are pending in the application. 
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DETAILED ACTION 

This office action is in response to the amendment filled August 21 , 2007. Claims 1-22 
are pending. Claims 5, and 13 canceled, claims 1,7-9, 12 and 19 are amended, and 
claims 20-22 are added. 



Response to Arguments 

Applicant's arguments filed August 21 , 2007 have been fully considered but they 
are not persuasive. 

In response to applicant's argument that the references fail to show certain 
features of applicant's invention, it is noted that the features upon which applicant relies 
(i.e., that Accessibility information is non-textual, semantic information that indicates 
non-verbal aspects of the textual communication, such as tone of voice and emotional 
content, for example (Remarks page 9)) are not recited in the rejected claim(s). 
Although the claims are interpreted in light of the specification, limitations from the 
specification are not read into the claims. See In re Van Geuns, 988 F.2d 1 181 , 26 
USPQ2d 1057 (Fed. Cir. 1993). 

Claim Rejections - 35 USC §112 

The following is a quotation of the second paragraph of 35 U.S.C. 112: 



The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 
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Claims 8,9,19,21 and 22 are rejected under 35 U.S.C. 112, second paragraph, 
as being indefinite for failing to particularly point out and distinctly claim the subject 
matter which applicant regards as the invention. 

Claim 8 recites, "wherein the translation step includes the use of automatic 
speech recognition of the first input and the second input to produce the electronic 
format, the first input including verbal information, and the second input including non- 
verbal information" however this presents a logical error. The second input, being non- 
verbal input, cannot be recognized by an ASR system, since ASR systems are 
designed specifically for the recognition of speech, i.e. verbal information, only. 

Claim 9 recites, "wherein said translation step includes the use of sign language 
recognition of the first stage input and the second stage input to produce the electronic 
format, the first Input including verbal information, and the second input including non- 
verbal information." however this also produces a logical error. A sign language 
recognition system is capable of recognizing gestures alone, not speech. 

In view of the aforementioned ambiguities, claims 8 and 9 are interpreted as 
"wherein said translating step includes the use of speech recognition of the first input, 
the first input including verbal information, and the second input including non-verbal 
information" and, "wherein said translating step includes the use of sign language 
recognition of the second input, the first input including verbal information, and the 
second input including non-verbal information", respectively. This interpretation used 
throughout the remaining of this office action. 
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Claim 19 recites the limitation "the speech input" and "the gestures input" in line 
5. There is insufficient antecedent basis for this limitation in the claim. 

Claim 21 recites, "wherein the same medium comprises automatic speech 
recognition" however this claim, which incorporates the limitations of claim 19, is 
ambiguous. The logical error associated with claim 8 also applies to this claim. The 
second input, being non-verbal input, cannot be recognized by an ASR system, since 
ASR systems are designed specifically for the recognition of speech, i.e. verbal 
information, only. In addition, the new limitation recited in claim 21 seems to contradict 
the second limitation of claim 19, which states, "translating the speech input using 
automatic speech recognition and the gestures input using sign language recognition". It 
is unclear as to whether the first and second input will be processed by the same 
medium or separate mediums, and to what order this processing occurs. 

Claim 22 recites, "wherein the same medium comprises sign language 
recognition", however this claim, which incorporates the limitations of claim 19, is 
ambiguous. The logical error associated with claim 9 also applies to this claim. A sign 
language recognition system is capable of recognizing gestures alone, not speech. In 
addition, the new limitation recited in claim 22 seems to contradict the second limitation 
of claim 19, which states, "translating the speech input using automatic speech 
recognition and the gestures input using sign language recognition". It is unclear as to 
whether the first and second input will be processed by the same medium or separate 
mediums, and to what order this processing occurs. 
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In view of tlie aforementioned ambiguities, claims 21 and 22 are interpreted as 
"translating the verbal input using automatic speech recognition" and "translating the 
non-verbal input using sign language recognition", respectively. This interpretation used 
throughout the remaining of this office action. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

Claims 1-17 and 19-22 are rejected under 35 U.S.C. 103(a) as being 

unpatentable over Ga/anes (2003/0130854). 

As per claim 1 , Galanes discloses a method of input conversion to output, said 
conversion method comprising: 

Entering a first input and a second input ([0053], multimodal input); 

Translating the first input and the second input into an electronic format ([0053] 
and [0049], multimodal input is recognized); 

Converting the electronic format into standard XML encoded with accessibility 
information ([0049], the XML is extended to include multimodal recognition functions 
(accessibility information)); 
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Galanes does not explicitly disclose transforming the standard XML encoded with 
accessibility information into individual version of XML dependent on desired output, 
and utilizing a rendering engine to modify the individual version of XML into a format for 
the output. However, Galanes does disclose utilizing a rendering engine to modify XML 
Into a format for the output ([0049] and [0056], recognized input is rendered as text for 
display). Galanes also states that the markup language extensions disclosed can be 
used with any other SGML-derived markup ([0050]). In addition, XML is a general- 
purpose markup language that Is extensible, i.e. It enables its users to define their own 
tags. This functionality enables users to share data across different information 
systems, or format data to reach a user defined purpose. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the Invention to transform the standard XML encoded with accessibility Information 
into individual version of XML dependent on desired output, and utilize a rendering 
engine to modify the individual version of XML into a format for the output In Galanes, 
since one of ordinary skill in the art has good reason to pursue the options within his or 
her technical grasp in order to achieve the predictable result of using available tools for 
modification of input data for a specific user defined purpose. 

As per claim 2, Galanes the conversion method of claim 1 , wherein said entering step 
includes speaking ([0027]). 
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As per claim 3, Galanes the conversion method of claim 1 , wherein said entering step 
includes gesturing ([0027]). 

As per claim 4, Galanes the conversion method of claim 1 , wherein said entering step 
includes writing ([0027]). 

As per claim 6, Galanes the conversion method of claim 1 , wherein said entering step 
includes semantic information ([0027], spoken input is entered). 

As per claim 7, Galanes the conversion method of claim 1 , wherein said entering step 
includes providing formatting and structure for text ([0027], in conjunction with speecli 
recognition, a pointing device is used, for example, to indicate the fields for data entry). 

As per claim 8, Galanes the conversion method of claim 2, wherein said translating step 
includes the use of automatic speech recognition of the first input to produce the 
electronic format ([0027]), the first input including verbal information ([0053], input 
speech), and the second input including non-verbal information ([0053], speech input in 
conjunction with multiple multimodal inputs). 

As per claim 9, Galanes the conversion method of claim 3, wherein said translating step 
includes the use of sign language recognition of the second input to produce the 
electronic format ([0027]), the first input including verbal information, and the second 



Application/Control Number: 10/686,127 Page 8 

Art Unit: 2626 

input including non-verbal ([0053], input speech), and the second input including non- 
verbal information ([0053], speech input in conjunction with multiple multimodal inputs). 

As per claim 1 0, Galanes the conversion method of claim 1 , however Galanes does not 
explicitly disclose wherein the individual version of XML is XHTML for desired output. 
However, Galanes does disclose that XHTML is a well-known markup language 
([0041]). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to use XHTML as the individual version on XML in Galanes, since one 
of ordinary skill in the art has good reason to pursue the options within his or her 
technical grasp in order to achieve the predictable result of using available tools for 
modification input data for a specific user defined purpose. 

As per claim 1 1 , Galanes the conversion method of claim 1 , however Galanes does not 
explicitly disclose wherein the individual version of XML is VoiceXML for desired output. 
However, Galanes does disclose that XHTML is a well-known markup language 
([0004]). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to use VoiceXML as the individual version on XML in Galanes, since 
one of ordinary skill in the art has good reason to pursue the options within his or her 
technical grasp in order to achieve the predictable result of using available tools for 
modification input data for a specific user defined purpose. 
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As per claim 12, Galanes the conversion method of claim 1 , wherein the accessibility 
information includes non-verbal information ([0049], the XML is extended to include 
multinnodal recognition functions (accessibility information)). 

As per claim 14, Galanes the conversion method of claim 1 1 , wherein the desired 
output is synthesized speech ([0049]). 

As per claims 15,16 and 17, Galanes the conversion method of claim 12, however 
Galanes does not disclose wherein the desired output is virtual sign language, 
electronic Braille, or electronic large print. However, Galanes does disclose that the 
system is functional for users with disabilities ([0028]). Galanes also discloses that the 
system includes devices such as a display screen and a touch sensitive pad ([0040]). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to have output in the form of virtual sign language, electronic Braille, or 
electronic large print in Galanes, since it would enable users with disabilities to more 
easily interact with the device. 

As per claim 19, Galanes discloses a method of input conversion to output, said 
conversion method comprising: 

Entering a first input including verbal information and a second input including 
non-verbal information, the second input being entered in parallel with the first input, the 
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second input being entered via a same medium as the first input, the same medium 
being one of a visual medium and an audial medium ([0037 and [0053], a computing 
device, whicli includes a camera and a microphone, receives multimodal input, 
including speech (verbal information) and information entered through the keyboard 
(non-verbal information)); 

Translating the speech input using automatic speech recognition and the 
gestures input using sign language recognition into an electronic format ([0053] and 
[0049], multimodal input, including speech and gestures, are recognized) 

Converting the electronic format into standard XML encoded with accessibility 
information ([0049], the XML is extended to include multimodal recognition functions 
(accessibility information)). 

Galanes does not explicitly disclose transforming the standard XML encoded with 
accessibility information into individual version of XML dependent on desired output, 
and utilizing a rendering engine to modify the individual version of XML into a format for 
the output. However, Galanes does disclose utilizing a rendering engine to modify XML 
into a format for the output ([0049] and [0056], recognized input is rendered as text for 
display). Galanes also states that the markup language extensions disclosed can be 
used with any other SGML-derived markup ([0050]). In addition, XML is a general- 
purpose markup language that is extensible, i.e. it enables its users to define their own 
tags. This functionality enables users to share data across different information 
systems, or format data to reach a user defined purpose. 
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Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to transform the standard XML encoded with accessibility information 
into individual version of XML dependent on desired output, and utilize a rendering 
engine to modify the individual version of XML into a format for the output in Galanes, 
since one of ordinary skill in the art has good reason to pursue the options within his or 
her technical grasp in order to achieve the predictable result of using available tools for 
modification of input data for a specific user defined purpose. 

As per claim 20, Galanes discloses a method of input conversion to output, said 
conversion method comprising: 

Entering a first input including text information and a second input including 
semantic information ([0037] and [0053], multimodal data is received, including 
information entered through a keyboard (text) and speech (semantic information)); 

Translating the first input and the second input into electronic format ([0053] and 
[0049], multimodal input is recognized); 

Converting the electronic format into standard XML encoded with accessibility 
information ([0049], the XML is extended to include multimodal recognition functions 
(accessibility information)); 

Galanes does not explicitly disclose transforming the standard XML encoded with 
accessibility information into individual version of XML dependent on desired output, 
and utilizing a rendering engine to modify the individual version of XML into a format for 
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the output. However, Galanes does disclose utilizing a rendering engine to modify XML 
into a format for the output ([0049] and [0056], recognized input is rendered as text for 
display). Galanes also states that the markup language extensions disclosed can be 
used with any other SGML-derived markup ([0050]). In addition, XML is a general- 
purpose markup janguage that is extensible, i.e. it enables its users to define their own 
tags. This functionality enables users to share data across different information 
systems, or format data to reach a user defined purpose. 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to transform the standard XML encoded with accessibility information 
into individual version of XML dependent on desired output, and utilize a rendering 
engine to modify the individual version of XML into a format for the output in Galanes, 
since one of ordinary skill in the art has good reason to pursue the options within his or 
her technical grasp in order to achieve the predictable result of using available tools for 
modification of input data for a specific user defined purpose. 

As per claim 21 , Galanes discloses the method of claim 19, wherein the verbal input is 
input using automatic speech recognition ([0037] and [0053], multimodal data is 
received, including information entered through a keyboard (text) and speech (semantic 
information)). 

As per claim 22, Galanes discloses the method of daim 19, wherein the non-verbal 
input is input using sign language recognition ([0037] and [0053], multimodal data is 
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received, including infornriation entered through a keyboard (text) and speech (semantic 
information)). 

ClaimIS is rejected under 35 U.S.C. 103(a) as being unpatentable over Galanes 
in view of Lemlouma ("NAC: A Basic Core for the Adaptation and Negotiation of 
Multimedia Services" 2001 ). 

Galanes discloses the method of claim 1 , however Galanes does not disclose whrein 
said transforming step includes the use of extensible stylesheet language 
transformations to transform the standard XML encoded with accessibility information 
into the individual version of XML dependent on desire output. However, XML is a 
general-purpose markup language that is extensible, i.e. it enables its users to define 
their own tags. This functionality enables users to share data across different 
information systems, or format data to reach a user defined purpose. In addition, 
Lemlouma discloses that extensible stylesheet language transformations, or style 
sheets, are commonly used for adaptation of XML models (section 1 Introduction, last 
paragraph). 

Therefore it would have been obvious to one of ordinary skill in the art at the time 
of the invention to the use of extensible stylesheet language transformations to 
transform the standard XML encoded with accessibility information into the individual 
version of XML dependent on desire output in Galanes, since one of ordinary skill in the 
art has good reason to pursue the options within his or her technical grasp in order to 
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achieve the predictable result of using available tools for modification of input data for a 
specific user defined purpose. 



Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of tNs final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Dorothy Sarah Siedler whose telephone number is 571- 
270-1067. The examiner can normally be reached on Mon-Thur 9:30am-5:30pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Richemond Dorvil can be reached on 571-272-7602. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 
273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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